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(54) Image processing method and device and scanner and printer equipped with same 



(57) A nnethod and apparatus to detect marks of a 
specified shape and color on a document and to extract 
a specified pattern consisting of a number of these 
marks. An RGB color signal is sent, color by color, by 
way of image input unit (12) to binary processing unit 
(13). If the marks are yellow, the B signal for the pixels 
constituting the marks will be smaller than the other sig- 
nals. The B signal is sent to mark shape extraction unit 
(13a) and threshold processing is executed. Simultane- 
ously, R. G and B are sent to mark color extraction unit 
(13b), and thefact that the marks are yellow Is detected. 
AND element (13c) finds the logical product of these 



processes, and a binary image is generated from which 
only the yellow pixels have been extracted. This binary 
image is transmitted to mark location detection unit (15) 
by way of storage unit (1 4). The shapes of the marks are 
matched against reference data and location data are 
extracted for the marks. The location data are transmit- 
ted to pattern location matching unit (17) by way of stor- 
age unit (16). The goodness of fit representing whether 
the marks are in specified locations is obtained and out- 
put. 
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Description 

FIELD OF THE INVENTION 

This invention concerns an image processing 
method and device optimally suited to prevent the read- 
cut or printing of documents which may not legally be 
copied, such as bank notes, negotiable securities or top 
secret documents, as well as a copier, scanner or 
printer in which it is installed. 

BACKGROUND OF THE INVENTION 

The development in recent years of copying 
devices such as full-color copiers has enhanced the 
quality of reproduction to a level at which one cannot 
distinguish the copy from the original with the naked 
eye. Such faithful reproductions can now be obtained 
inexpensively. However, along with this benefit comes 
the increased risk of copiers being put to such illicit uses 
as counterfeiting money, negotiable securities, and 
other original documents whose reproduction is legally 
prohibited, or the purloining of top secret documents by 
copying them. To avert this danger, various devices 
have been developed to prevent copiers from being 
used for counterfeiting. One such device is the image 
processing device disclosed in Japanese Patent Publi- 
cation 2-210591. 

That image processing device is installed in a full- 
color digital copy machine which reads the original 
image placed on the glass of the copier by scanning the 
entire original four times and executes the processing 
needed to copy it. Let us assume a bank note has been 
placed on the glass of the copy machine. On the first 
scan, the device detects the rough location of water- 
mark B, the mark on bank note A which identifies it as 
money (see Figure 37). the item which the machine is to 
be prevented from copying, on the assumption that the 
document on the glass is. in fact, a bank note. 

On the second scan, the device uses the location of 
watermark B to detect the exact position of the bill and 
the angle at which it is placed (I.e., the exact position 
coordinates of the bill on the glass). Because the length, 
size and shape of bank note A. the item to be detected, 
are already known, we can extract the edge C (the 
hatched portion) of bank note A by binarizing the image 
data at a given threshold. Using this edge, we can 
obtain the coordinates (x^ . y^) and (xg, ya) of two apices 
D. 

In this way we can obtain the slope theta at which : 
the bill is oriented. On the third scan, we use the exact 
position of the bill (the coordinates of apices D and 
angle theta) obtained in the second scan to calculate 
the position coordinates (xg, y^) of the red seal E 
stamped on the bill. Based on these position coordi- t 
nates, we extract an image from the region in which the 
seal is placed, and we make a judgment as to whether 
there is in fact a red seal on this item. 

By scanning the item three times, the device is able 



to determine whether or not a document which it is sup- 
posed to detect (in this case, a bank note) is loaded in 
the copier. If the device detects that a prohibited docu- 
ment such as a bank note has been loaded to be cop- 
5 ied, it will perform its specified anti -counterfeiting 
operations on the fourth scan, such as making the entire 
display go black or preventing the copier from operating. 

The existing device described above requires an 
extremely large memory capacity merely to store data 
10 concerning the size of paper money Inevitably, then, 
only a limited number of kinds of money can be 
detected. 

Furthermore, three scans are needed to determine 
whether the item on the glass is a non-reproducible doe- 
rs ument such as a bank note. If the apices D of bank note 
A are folded down or mutilated in some other way the 
device will be unable to determine the location of red 
seal E. and consequently will be unable to detect that 
the item being processed is a bank note, one of the 
20 items it is supposed to detect. 

One solution to this problem would be to surround 
specified pattern E with a guideline F consisting of 
either a heavy continuous line or a broken line, as 
shown in Figures 38 (A) and (B). The region Inside 
25 guideline F could then be extracted to determine if it 
contains the specified pattern. If the type of guideline F 
shown in the drawing is used. then, it will be easier to 
separate the region containing the specified pattem 
from the other image data. The straight portions of 
30 guideline F can be used to calculate the slope and angle 
of orientation of the bill as well as the coordinates of the 
apices. The use of a guideline thus allows us to extract 
the specified pattern more easily and accurately 

When this method is used, however, guideline F is 
35 conspicuous on the bill (because it is a large feature). It 
can easily be seen and recognized, as can the specified 
pattern E which it encloses. This makes it easier for 
someone to tamper with guideline F or specified pattem 
E (by adding extra lines or erasing portions of lines). If 
40 the pattern or guideline has been tampered with, it is 
possible that the pattern will not be recognized (or 
extracted), or that even if it is recognized it will not be 
determined to be the right pattern. 

Another problem is that at least two scans are 
45 required to detect the location of bank note A and guide- 
line F. find the location of the specified pattern, extract 
the pattern and determine whether it is the right pattern. 
The device could not then be used in copy machines 
which read the image in a single scan. If we wish to 
?o extract the specified pattern without any preliminary 
scans, our only recourse is to match the pattern against 
a reference as each pixel is read in the raster direction. 
Since we do not know the orientation of the pattern (i.e. . 
the angle at which the image is oriented on the glass), 
s we must also consider the angle of rotation when we 
match the pattern. This makes our task extremely diffi- 
cult. 

The existing method or the method using guideline 
F required selecting from the many patterns on a non- 
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reproducible document a pattern suitable for feature 
analysis (i.e., one which can be discriminated accu- 
rately and which is not found on any document which 
may be legally copied), such as the red seal on a bank 
note, and matching it against a reference pattern. Even 
if we limit our focus to Japan, there are many, many doc- 
uments which may not legally be copied. To detect all of 
them reliably would require a huge memory capacity 
and lengthy processing. Existing devices, then, lack the 
capacity to execute the required processing in real time. 
Furthermore, such devices require three scans to deter- 
mine that the document may not be copied. This too 
stands in the way of high-speed processing. 

If, after the copier is manufactured, a new docu- 
ment appears which may not be copied, the device will 
not be able to handle the document or even detect it. To 
detect the document, a new set of features must be 
selected and taught to the copier, which is a compli- 
cated procedure. If we are to include the documents of 
other countries, the problem becomes even more over- 
whelming. 

SUMMARY OF THE INVENTION 

The objective of this invention is to provide an 
image processing method arxi device which solves 
these various problems as well as a copier, scanner and 
printer in which they are employed. More specifically, if 
there are marks or a specified pattern composed of 
marks on a document, this method and device would be 
able to execute the specified processing (actually deter- 
mining whether this is the right pattern, extracting the 
data needed for the determination, and so on) required 
to detect this pattern unmistakably and evaluate it easily 
and reliably 

Another objective of this invention is to provide a 
method and device such that the specified pattern 
would be difficult for an uninformed person to discern, 
so as to minimize the probability of tampering, as well 
as a copy machine or other device in which that method 
and device are employed. 

Yet another objective is to provide a method and 
device which would be able to recognize the specified 
pattern on the first scan, which would be able to recog- 
nize shapes using a minimal hardware configuration 
and a simple algorithm, and which would be capable of 
high-speed processing, as well as a copy machine or 
other device in which that method and device are 
employed. 

To achieve the objectives outlined above, the image 
processing method of this invention is designed as fol- 
lows. With this method, the image is processed to 
detect in the received image data a number of marks of 
a given color or shape. The aforesaid marks are 
detected (by threshold processing, in the embodiment) 
using only one of the color signals constituting the color 
image. If the aforesaid color signals are received suc- 
cessively, color by color, in the order in which they 
appear in the image, it Is desirable that the color signal 



which will be used to detect the aforesaid marks should 
be received first and processed in specified fashion. 

The aforesaid marks may be detected using data 
obtained by performing threshold processing on the sin- 

5 gle color signal mentioned above as well as data 
obtained by extracting all pixel data for color signals dif- 
ferent from the aforesaid color signal whose density is 
within a given range. When the marks are detected 
using the method in above, the density data for the 

10 region where the marks are found may be extracted and 
the goodness of fit of these data may be obtained with 
respect to a previously established set of color data. 
The goodness of fit of these data with respect to the 
color data and their goodness of fit with respect to a 

15 specified pattern defined by the aforesaid shape may be 
output either separately or in combination. The given 
color of the pixels constituting the aforesaid marks may 
be yellow. 

In an image processing method such that a number 

20 of marks of a given color and shape are detected in 
received linage data and the spatial relationship of 
these marks is then used to find their goodness of fit 
with respect to a specified pattern, the aforesaid marks 
may be detected using one of the methods described in 

25 above. In such a case, the aforesaid marks may be 
located on the peripheries of a number of given shapes 
(squares, in the embodiment). When the goodness of fit 
of the marks with respect to the aforesaid specified pat- 
tern is obtained, the angle of orientation of the aforesaid 

30 Specified pattern may be obtained based on data repre- 
senting the marks on the periphery of one of the afore- 
said shapes (in the embodiment, one of the two 
squares). This angle may then be used to extract for ref- 
erence purposes the data representing the marks on 

35 the peripheries of the other shapes, and these refer- 
ence data may be compared with previously collected 
data. 

A device ideally suited to implementing any of the 
methods described above would be an image process- 

40 ing device capable of detecting a number of specified 
marks in received image data. This device has the fol- 
lowing components: a binary processing device which is 
equipped, at the very least, with a shape extraction unit 
to execute threshold processing on one of the color sig- 

45 nals comprising a color image; a storage device to store 
the binary image output by the binary processing 
device; and a device to detect the location of the marks, 
which reads out the binary image stored in the storage 
device and matches the marks in it against the pattern 

so which is to be detected. 

The aforesaid binary processing device should ide- 
ally have the following: a shape extraction unit; a color 
extraction unit which extracts, at minimum, all pixel data 
representing signals for colors other than the aforesaid 

55 color whose density is within a given range; and an AND 
element which receives the output of the aforesaid 
shape extraction and color extraction units. 

This image processing device should also have a 
device placed parallel to the aforesaid binary process- 
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ing device to store the aforesaid received image data, 
and a device which receives the output of the aforesaid 
device to extract the locations of the marks and extracts 
from the density data associated with the color signals 
for the portion of the image where the marks are 5 
detected the goodness of fit of the color (in the embodi- 
ment, extraction device 20. averaging unit 21 and 
matching unit 22). 

The image processing device should also have a 
device to store data representing the locations of the 10 
marks detected by the aforesaid device for that pur- 
pose; and a device which uses the location data stored 
in the aforesaid device to match these marks against a 
specified pattern consisting of a number of marks. 

The copy machine, scanner or printer of this tnven- 15 
tion is equipped with any one of the image processing 
devices described above. The image data output by the 
aforesaid device which reads the document loaded in 
the copy machine are input in parallel into both the 
device to convert the color signals and the aforesaid 20 
image processing device. Input and output signals to 
the control device in the scanner or printer are input in 
parallel to the image processing device. 

The image processing device executes the 
processing required to detect the aforesaid marks or 25 
specified pattern using image data obtained during the 
specified scan. It decides whether the aforesaid marks 
or specified pattern appear on the document being 
processed. If it determines that they do, it transmits a 
control signal to the aforesaid processing device and 30 
controls the specified processing, whether copying, 
reading the image, or output (i.e., print-out). 

To detect (or recognize the shape of) a number of 
marks of a given color and shape, threshold processing 
is executed for a single specified color. Choosing an 35 
appropriate color causes the signals for that color to 
have a high density among the color signals (RGB. 
YMC. Yab, ...) of which the color image is constructed. If 
yellow is chosen, for example, the density of the B, Y 
and b signals will get higher (their numerical values will 40 
get smaller), and a large difference will be expressed 
between their values and the values of the background 
color (the B, Y and b signals) surrounding the marks. By 
executing threshold processing on a given signal, then, 
we can separate the marks from the background; and 45 
based on this, we can recognize the pattem. 

When the signals are transmitted color by color in 
the order of how much surface they occupy in the 
image, and the color signal used to recognize the pat- 
tern is the first one transmitted, the determination of so 
whether there are any marks can be made before the 
transmission of the other color signals is completed 
(i.e., before the image is completed). If there are marks, 
the processing can be halted at this time. 

While the marks are being extracted, a judgment is 55 
made in parallel with the extraction based on a single 
color signal discussed above as to whether the density 
of the remaining color signals is within a given range. By 
finding the logical product of the two processes, we can 



generate a binary image which can be used to finally 
extract the marks. This is helpful because when a single 
color signal is used, it is posstole that pixels of that color 
outside the marks will also be extracted in the extraction 
processing. However, even if this happens, the probabil- 
ity that the marks will not be recognized is extremely 
small, since the spurious pixels will be removed in the 
final pattern recognition process. When a binary image 
is generated, pixels which are not part of the marks will 
go to "0", so the marks can be extracted with greater 
accuracy. 

When the image processing device is configured as 
described in this invention, the goodness of fit of the 
color of the marks is extracted using only the density 
data, and this is done in parallel with the extraction of 
the pattern. Extracting the goodness of fit of the color 
does not entail any other judgments, so using this 
method has the effect of reducing the aggregate 
processing time. If marks are detected, the density of 
the region containing the marks is obtained and 
matched against previously assembled color data. 
Using both the goodness of fit of the pattern and that of 
the color to make a judgment significantly reduces the 
probability that the marks (and so the specified pattern 
that they form) will not be detected when either the pat- 
tern or color has been tampered with. 

When yellow is used as the constituent color of the 
marks, the density of at least one color signal will be 
high no matter whetiier the signals are BRG, YMC, Yab 
or some other system. Using yellow thus allows the 
device to be used in equipment with various image for- 
mats. Yellow is also a color which is difficult for the 
human eye to discriminate, so the marks will be difficult 
to find, which makes it harder to tamper with tfiem. 

According to the invention disclosed in this inven- 
tion, the pattern to be detected is composed of a 
number of marks which are arranged according to a 
previously established rule (viz.. along the contour of a 
given shape). When tiie marks are extracted using any 
of the various methods and methods described above, a 
judgment can be made as to whether the marks along 
the contour of the aforesaid shape are arranged in a 
given way. In tiiis way we can obtain the goodness of fit 
of the marks with respect to the specified pattern. If tfie 
goodness of fit is high, we can conclude tiiat the marks 
do constitute the specified pattern, and specified 
processing can be executed to halt the copying, read- 
out or output operation. This processing may include 
causing related processing to cease or executing a 
given process on the original image, such as printing 
out a black screen. This is true for the following discus- 
sion as well. Since we know that the marks are 
arranged along the contour of a given shape, we can 
use a window appropriate for that shape to search for 
the marks when we need to determine whether they are 
arranged in a given way. This will make it easier to 
detect the marks. 

With the device described in the emtxxliment, the 
specified pattern is composed of a number of marks 
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arranged along the contours of a number of shapes. 
The locations of the marks on the contour of at least one 
of the shapes are detected. If there are no marks along 
the outlines of the other shapes, we can determine that 
this is not the specified pattern. If the marks are in a 
given spatial relationship, there is a high probability that 
this is the specified pattern. Using the absolute coordi- 
nates of the marks to detect this relationship allows us 
to determine the orientation (i.e.. the angle of rotation) 
of the pattern. The locations where the marks along the 
other shapes should be found at that angle of rotation 
can be used as reference data. By comparing the actual 
location data we obtain with these hypothetical location 
data, we can easily obtain the goodness of fit of the 
marks to the pattern. 

When a copy machine equipped with the image 
processing device of this invention is used to try to copy 
a bank note or other non-reproducible document, or 
when a scanner or printer equipped with this device is 
used to try to read or print out such a document, the 
specified pattern on the document can be detected. A 
command can be issued to halt the copying operation, 
and the reproduction, read-out or print -out of facsimiles 
of the document can be prevented. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows an example of marks and a pattern 
which might be used in this invention. 

Figure 2 illustrates the principle by which the marks 
used in this invention are arranged. 

Figure 3 shows an example of real and dummy 
marks and a pattern which might be used in this inven- 
tion. 

Figure 4 shows another example of real and 
dummy marks and a pattern which might be used in this 
invention. 

Figure 5 shows the overall configuration of the first 
Ideal embodiment of the image processing device of 
this invention. 

Figure 6 shows the internal configuration of the 
shape extraction unit. 

Figure 7 shows the internal configuration of the 
color extraction unit. 

Figure 8 illustrates the operation of the binary unit. 

Figure 9 illustrates the operation of the binary unit. 

Figure 10 illustrates the operation of the binary unit 

Figure 1 1 illustrates the operation of the binary unit. 

Figure 12 illustrates the operation of the binary unit. 

Figure 13 illustrates the operation of the binary unit 

Figure 14 shows an example of a pattern which 
might be stored in storage device 14. 

Figure 15 shows the window used in the location 
detection unit. 

Figure 16 illustrates the state of the data stored in 
storage device 16. 

Figure 17 shows the internal configuration of the 
location detection unit. 

Figure 1 8 shows an example of a window unit which 



might be used in the location detection unit. 

Rgure 19 shows an example of a decoder circuit 
which might be used in the location detection unit. 

Rgure 20 illustrates how the data are stored in stor- 
5 age device 16. 

Figure 21 shows the internal configuration of the 
arrangement matching unit 

Figure 22 illustrates the concept used in the window 
element of the arrangement matching unit. 
10 Rgure 23 shows the actual hardware constituting 
the window element. 

Rgure 24 illustrates the principle underiying the 
organization of the output of the window element. 

Figure 25 shows an example of the data latching 
15 circuit used to compare the marks on the outer squares. 
This circuit is a component of the arrangement match- 
ing unit 

Rgure 26 shows an example of the data latching 
circuit used to compare the marks on the inner squares. 
20 This circuit is a component of the arrangement match- 
ing unit. 

Rgure 27 shows examples of outputs of the pattern 
window. 

Rgure 28 shows data and addresses stored in stor- 
es age unit 1 7. the memory for the outer square. 

Rgure 29 shows an example of a comparison cir- 
cuit which could be a component of the arrangement 
matching unit. 

Figure 30 shows the overall configuration of the 
30 second ideal embodiment of the image processing 
device of this invention. 

Figure 31 illustrates the function of the density 
matching unit. 

Rgure 32 shows the overall configuration of the 
35 third ideal embodiment of the image processing device 
of this invention. 

Rgure 33 shows an example of a copy machine 
designed according to this invention. 

Rgure 34 shows an example of a copy machine 
40 designed according to this invention. 

Figure 35 shows an example of a scanner designed 
according to this invention. 

Rgure 36 shows an example of a printer designed 
according to this invention. 
45 Rgure 37 illustrates the operation of an image 
processing device belonging to the prior art. 

Rgure 38 shows two attempts to address the prob- 
lem in the prior art. 



In this section we shall provide, with reference to 
the appended drawings, a detailed explanation of ideal 
embodiments of the image processing method and 
55 device of this invention as well as the copier, scanner or 
printer in which they are employed. We shall begin by 
using Figures 1 through 4 to explain an ideal embodi- 
ment of the image processing method of this invention. 

In this first embodiment, a pattern such as that 
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Shown in Figure 1 is provided in a given location on the 
document to be detected, which is something that may 
not legally be copied. A determination is made as to 
whether pattern 1 can be found in image data which are 
captured and transmitted. H it can. a signal representing 
detection is output. 

Pattern 1 is composed of a number of marks 2 
which are placed in appropriate locations. In this 
ennbodiment, marks 2 are small triangles which form the 
shape of a frame with nothing in the center. These trian- 
gles are of a specified color. The principle underlying 
the an-angement of marks 2 in this example is shown in 
Figure 2. The marks are placed along the sides of two 
squares. In the example shown, there is a single mark 2 
In a specified location on L^, the line defining the inner 
square, and ten marks 2 in specified locations on L2. the 
line defining the outer square. When marks 2 are placed 
along the sides of squares (or rectangles) in this way. 
pattern 1 can be printed so that the sides of the square 
are parallel to the horizontal and vertical axes of the 
image. The marks will then correspond to the scanning 
path when the image is read, virfiich will simplify the 
detection hardware by allowing a smaller window to be 
used. When pattern 1 is actually printed, of course, lines 
Li and L2 will not be there, and only the marks 2 shown 
in Figure 1 will be visible. This will make it difficult for an 
uninformed person to figure out the principle of the 
arrangement and so make it more difficult to tamper 
with the pattern. 

If the coordinate position of each mark 2 is speci- 
fied, the marks can be made quite small (in comparison 
with the guidelines used in the prior art) . This will make 
it difficult for an uninformed person to find the marks but 
will not affect the recognition rate. Using smaller marks 
also allows us to use a smaller window to detect the 
marks. A simpler arxJ cheaper circuit can be used to 
detect the marks, and the processing speed can be 
increased. 

According to this invention, all of marks 2 are 
formed using a given color, as discussed above. The 
marks may be of identical or similar colors. The color is 
one with a high density in one or the three color signals 
expressing the color data in an RGB or other format. 

With this embodiment of the image processing 
method, a pattern 1 with the characteristics set out 
above is provided in a given location in the image we 
wish to detect. When an image Is read and output, the 
image data are scanned and the resemblance of any 
part of that image to pattern 1 is obtained. If the afore- 
said marks are added to the image before a new bank 
note is Issued so that marks 2 and pattern 1 can be 
detected, this system, despite its small memory, will be 
able to detect any type of bank note or other non-repro- 
ducible document. 

In this embodiment, pattern 1 is composed of a 
number of marks 2. The first step is to detect these 
marks 2 in the image and determine their locations. The 
density of one of the three color signals constituting the 
marks will be higher than that of the other two. By exe- 



cuting threshold processing in the image signals with 
respect to this color signal and binarizing the image 
data, we can extract marks 2 and recognize the pattern. 
In this embodiment, we can achieve greater accu- 
5 racy by extracting the pixels constituting marks 2. In 
addition to pattern recognition t>ased on a single color 
signal, we can determine whether each pixel in the mark 
has a given color density. When a single color signal is 
subjected to threshold processing independently of the 
10 other signals, it is possit>le for another color (i.e.» a color 
different from the color of the marks) with a density 
above the threshold to be extracted as well. For this rea- 
son, the image data obtained are filtered with respect to 
the density of the three color signals. By extracting pix- 
15 els whose density is within a given range, we can 
extract only pixels of the desired color. 

We can execute pattern recognition using a single 
color signal and then determine whether the pattern Is 
the specified color. By finding the logical product of 
20 these two processes, we can accurately extract just the 
pixels constituting the marks. We then determine 
whether marks of a given shape are found in the binary 
image conrposed of the pixels extracted In this way This 
concludes the detection process. 
25 We next obtain the locations of the marks we have 
extracted. By determining whether they are in a given 
spatial relationship with each other, we can find how 
closely they resemble the specified pattern (i.e.. their 
goodness of fit). Since the marks 2 which constitute pat- 
30 tern 1 in the example shown in Figure 2 are placed on 
the outlines of two squares, two windows can be set up 
which correspond to these squares. We could then 
determine whether, for example, the marks 2 found in 
these windows while scanning in the raster direction are 
35 in a given spatial relationship with each other. 

If, in addition to pattern 1 , dummy marks are printed 
in appropriate places as shown in Figures 3 and 4, it will 
be more difficult to tamper with the pattern. In Figure 3, 
dummy marks 3 of the same shape and color as marks 
40 2 (marked with an X in the drawing) have been placed in 
locations not on the outlines of the two squares dis- 
cussed above. Dummy marks 3 will be extracted as 
candidates for pattern 1 in the first detection process 
described above; but they will be eliminated by the next 
45 process, the spatial relationship test using the two win- 
dows. The dummy marks will not produce false posi- 
tives. 

An uninformed person will have no way of knowing 
which of the marks are the actual marks 2 constituting 

50 pattern 1, which are essential for pattern recognition, 
and which are the dummy marks 3. (In Figure 3. the 
dummy marks have been identified with an X for con- 
venience; In an actual situation, they would be of the 
same shape and color as marks 2. and would not be 

55 marked with an X.) This would make it very difficult for 
someone to tamper with the actual marks 2 only 

In the example in Rgure 4, dummy marks 4 of the 
same shape as marks 2 (indicated by hatching in the 
drawing) have been placed on tine outlines of the afbre- 
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said squares. These dummy marks are of a different 
color from the actual marks 2. In the first-stage detec- 
tion process in the processing method we have been 
discussing, the dummy marks will be eliminated when 
the data are binarized and/or when the pixels of a given 
color are extracted for pattern recognition, and they will 
not be extracted. They will not be erroneously recog- 
nized even though they are placed on the outlines of the 
squares. 

Since in this example marks 2 and 4 differ only in 
their color, an uninformed person will have no way of 
knowing which of the marks are the actual marks 2 con- 
stituting pattern 1 . which are essential for pattern recog- 
nition, and which are the dummy marks 4. This would 
make it very difficult for someone to tamper with the 
actual marks 2 only. 

Although this alternative is not pictured in the draw- 
ings, the same result can be achieved by placing trian- 
gles of different dimensions from marks 2 or dummy 
marks of other shapes around marks 2. By using an 
appropriate combination of different types of dummy 
marks, we can increase their effectiveness. 

Any color which meets the conditions set out above 
(i.e., its density in a given color signal must be high) may 
be used for the aforesaid marks. However, yellow, cyan 
and magenta are good choices, as their density will 
become higher (i.e., the numerical value of the signal 
will be smaller) in a single color signal in the position of 
an RGB signal. If yellow is used for the marks, its den- 
sity in a given signal (a B, Y, b signal) will increase 
regardless of whether the signal format is RGB, Lab or 
YMC. The pattern will be recognized accurately, but will 
be relatively difficult to discriminate with the naked eye. 
Even the fact that marks are there will not necessarily 
be noticed. 

In the method used in the first embodiment as 
described above, when the marks are extracted and 
when the mark is extracted using a single color signal, 
the image is binarized using either the other color sig- 
nals or all of the color signals. The mark is extracted, the 
locations of the marks are obtained using these data, 
and the goodness of fit is obtained. However, the inven- 
tion is not limited to this method only. It would also be 
possible, in addition to finding the goodness of fit of the 
mark, to focus separately on the density of each color 
signal. The density of the portions with the marks could 
be compared with previously assembled reference data 
to find the goodness of fit of the color, and the good- 
nesses of fit of the mark and the color can be combined 
to determine whether the specified pattern has been 
found. (This is the method used in the second embodi- 
ment.) The goodness of fit of the pattern can also be 
found using only the binary data obtained by extracting 
the mark based on a single color signal. (This is the 
method used in the third embodiment.) 

We shall next discuss an embodiment of the image 
processing device of this invention which is ideally 
suited for implementing the processing method 
described above. Figure 5 shows the overall configura- 
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tion of such an image processing device. In this exam- 
ple, the device is installed in a full-color copy machine. 
When someone uses the copy machine to try to copy a 
non-reproducible document such as a bank note, the 

5 image processing device detects this and interrupts the 
copying process. As can be seen in the drawing, the 
image data read by the image sensor in the copy 
machine are transmitted to image input unit 12 (a buffer 
IC) in image processing device 10. As these image data 

10 are scanned by an image sensor such as a CCD, they 
are transmitted successively in real time, region by 
region, to the image processing device. The actual data 
which are sent are 8-bit color data for each of the red 
(R). green (G) and blue (B) components. 

15 The RGB color signals pass through image Input 
unit 12 and are transmitted to binary processing unit 13. 
The binarized image data (i.e., a binary image) are 
stored in storage device 14. The binary image stored in 
device 14 is sent to mark location detection unit 1 5. The 

20 marks 2 constituting pattern 1 are extracted from the 
binary image, their locations are specified, and they are 
stored in storage device 16. Everything before device 
16 constitutes the aforesaid unit to detect the marks. 
The data representing the locations of marks 2 

25 which are stored in device 1 6 are transmitted to pattern 
location matching unit 17. A window of a specified 
shape and size is used to determine how well the loca- 
tions of the marks match a specified arrangement (i.e., 
the arrangement constituting pattern 1 (see Figure 1)) . 

30 The result of this determination is transmitted to good- 
ness output unit 18. If the goodness of fit received by 
goodness output unit 18 exceeds a given value, a signal 
indicating that the pattern has been detected is output 
to the copy machine. 

35 We shall next discuss the function and configura- 
tion of each component of the device. Binary processing 
unit 13 consists of mark shape extraction unit 13a, mark 
color extraction unit 13b and AND element 13c, which 
finds the logical product of the outputs of units 13a and 

40 13b. A single color signal with a high density, whichever 
of the RGB signals is best suited to extracting the mark, 
is sent to mark shape extraction unit 13a. The other 
color signals, or, alternatively, all the color signals, are 
sent to mark color extraction unit 13b. The marks we 

45 detected in this example are yellow, so the B signals are 
sent to mark shape extraction unit 13a. 

Mark shape extraction unit 1 3a, shown in Figure 6, 
executes threshold processirtg. It has a comparator with 
eight bits to match the eight bits constituting the color 

so signals. The B signal received from image input unit 12 
is input into input terminal a: a fixed threshold value is 
input into input terminal b. When a is less than b. the 
output will be "1". (The higher the density, the lower the 
numerical value.) In this embodiment, the threshold 

55 value is fixed. The use of a floating binary circuit would 
allow the threshold value to be changed according to 
the density of the document. Such a configuration would 
yield a more accurate pattern. 

Processing each RGB signal, mark color extraction 
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unit 13b. shown in Figure 7, extracts any pixel whose 
density falls within a fixed range (between an upper and 
a lower limit value). Unit 13b consists of three 4-bit win- 
dow comparators which input RGB signals one by one 
and compare them with upper and lower limit values 
(when b > a > c, the output will be "1 ") and a three-out- 
put AND element which finds the logical product of the 
outputs of the three comparators. The upper and lower 
limit values used by the comparators have a specified 
margin with respect to the RGB signal expressing the 
color of the marks 2 which are to be detected. When a 
pixel is found for which each of the R, G and B compo- 
nents has a density within the fixed range, the output of 
the AND element will go to "1", and the pixel will be 
extracted as a prospective constituent of a mark. 

Mark color extraction unit 13b uses a 4-bit window 
comparator because printed materials in general suffer 
from extreme non-uniformity (variation) of color. Higher 
color resolution would be pointless. However, as pattern 
recognition requires accuracy, mark shape extraction 
unit 13a uses an 8-bit comparator, as mentioned above. 
Separating pattern recognition from color extraction in 
this way allows us to extract the mark accurately and 
absorb errors due to variation among the colors consti- 
tuting the mark so that the mark can be specified cor- 
rectly 

The color signal used for the determination made 
by mark shape extraction unit 13a (the B signal) should 
not be used for the determination made by mark color 
extraction unit 13b. Rather, the R and G signals should 
be input into two 4-bit window comparators, and the out- 
put of these two comparators should be sent to a dual- 
input AND element. This method allows the circuit con- 
figuration to be simplified. It will frequently happen that 
the outputs of the window comparators processing the 
B signal in mark color extraction unit 13b and the mark 
extraction unit will both be T. When the density of a 
specified signal such as yellow is high, there will be 
many occasions when the density value reaches the 
upper or lower limit value set in the window comparators 
(FF... or 00...). On these occasions, the determination 
processing will be virtually identical to the comparator 
determination shown in Figure 6. Thus eliminating the 
determination processing in such cases will not appreci- 
ably affect the accuracy of detection. 

The aforesaid binary processing unit 13 shown in 
the drawing is designed to extract marks 2 of a single 
color (including similar colors which fall in the range 
extracted by unit 13b). If a number of different colors are 
used for marks 2. the appropriate number of mark color 
extraction units 1 3b could be provided along with mark 
shape extraction unit 13a and connected in parallel to 
image input unit 12. If completely different colors are 
used, a system of binary processing units 13 could be 
provided such that all outputs are sent to an OR ele- 
ment which would find their logical sum and generate a 
final binary image. 

If the densities of the color signals used to recog- 
nize the mark are virtually identical and those of the 



other color signals are different, a single mark shape 
extraction unit 13a could be used while a number of 
mark color extraction units 1 3b are systematized so that 
the outputs of ail units 13b are connected to and OR 
5 element. The output of this OR element and tiiat of 
mark shape extraction unit 13a would be connected to 
an AND element. With such a configuration, the actual 
marks 2 could be several different colors and the 
dummy marks 4 could be still other colors. This would 
10 make it extremely difficult for an uninformed person to 
figure out which of the marks are meaningful. If the 
image processing device contains a binary processing 
unit 13 like that described above and/or an appropriate 
combination of constituent circuits and elements, it will 
75 be able to extract the pixels in the real marks 2 accu- 
rately, and its recognition rate will not suffer. 

We shall now discuss the underlying principle used 
by the aforesaid binary processing unit 13. Let us con- 
sider a case in which ti'iangular marks 2 (referred to 
20 below as "the portions enclosed by two triangles") are 
printed with yellow ink as shown in Figure 8. If the other 
portion of the image is white, the portions where marks 
2 are printed will absorb light of the B signal but will 
reflect light of the G and R signals. The rest of the image 
25 will reflect all of the RGB signals. Thus the reflectivity of 
the RGB signal with respect to the direction indicated by 
the dotted line in Rgure 8 (A) would be such that B 
would be absorbed in the portions where marks 2 are 
found, as shown in Rgure 8 (B). The B signal would 
30 have a large amplitude and the other color signals 
would maintain a state of high reflectivity. Since tiiis 
untreated reflectivity corresponds to the numerical 
value of the color signal, we can extract only those pix- 
els which constitute marks 2 by setting an appropriate 
35 tiireshold value for the B signal. 

In the same way, if cyan ink is used, R will be selec- 
tively absorbed. The reflectivity of tiie pixels, as is 
shown in Figure 9. is such that only the R signal 
changes significantly If magenta ink is used. G will be 
40 selectively absorbed, and the reflectivity, as is shown in 
Figure 10. is such that only the G signal changes signif- 
icantly. If the marks are printed in cyan, then, the R sig- 
nal should be input into the aforesaid mark shape 
extraction unit 13a. If they are printed in magenta, the G 
45 signal should be input. 

If marks 2 are printed in blue, green or red ink, two 
colors of light (the two which are not the color of the ink) 
will be absorbed, as is shown in Figures 1 1 through 13. 
If the marks are blue, the amplitude of the R and G sig- 
50 nals will increase (Figure 11); if tiiey are green, the 
amplitude of the B and R signals will increase (Figure 
12): if tfiey are red, the anrplitude of the B and G signals 
will increase (Figure 13). Thus if the marks we wish to 
detect are blue, eitiier the R or the G signal should be 
55 sent to mark shape extraction unit 13a. as we can sur- 
mise from Figure 1 1 . When the image is binarized, the 
mark will stand out from the background. 

To extract magenta marks using the method 
described above, tiie G signal should be binarized. 
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However, this will also enable the simultaneous extrac- 
tion of blue and red marks (see Figures 1 1 and 13). To 
extract magenta marks only (when red and blue marks 
are not used or are used only as dummies), mark color 
extraction unit 1 3b should have only one system and the 
upper and lower limit values used in the window compa- 
rator should be set high for the R and B signals and low 
for the G signal. This will insure that blue and red are not 
extracted by mark color extraction unit 13b. Only 
magenta marks will appear in the binary image output 
by AND element 13c. If we wish to extract red or blue 
marks as well, mark color extraction unit 13b must have 
as many systems as there are colors to extract. Upper 
and lower limit values for each color should be set so 
that the range includes the density needed to extract 
that color. 

A binary processing unit 13 configured as 
described above outputs a binary image in which pixels 
which are the same color as mark 2 are expressed as 
"1" and pixels of other colors are expressed as "0". This 
image is stored in storage device 14, which in this 
embodiment consists of a given number of line buffers. 
Location detection unit 15 can use the aforesaid binary 
image to detect marks 2 and extract their locations. 
Once the locations of the marks are known, there is no 
need to store all the image data representing the docu- 
ment. Only a few dozen lines of buffer space are needed 
to store the data needed for subsequent processing. 

An example of a binary image which would be 
stored in device 14 (the line buffer) is shown in Figure 
14. Each square in the drawing represents one pixel. In 
the example we have been using, marks 2 are triangles. 
In order to make them distinguishable from dirt 
smudges, we have left them white in the center. The 
mark shown in Figure 14 is a minimal triangle. In this 
embodiment, the marks have been made as small as 
possible to enable the hardware needed to detect them 
to be made smaller as well as to make it more difficult 
for an uninformed person to find them. The mark of pix- 
els which constitutes the actual mark is that shown in 
Figure 14. 

Mark location detection unit 15 reads out the data 
stored in storage device 14 and detects the marks of 
pixels which constitute triangles like the one shown in 
Figure 14. In this example, it uses a window of 5 x 4 pix- 
els (Qoo through Q34) like that shown in Rgure 15. 
When the binary data for pixels Q02. On. Q13 and Q20 
through Q24 are "1" and those for the other pixels are 
"0'\ the HIT output goes to "1 and the location data are 
stored in storage device 16. 

The result or this determination (mark has been 
found: 1 ; no mark has been found: 0) is stored in device 
16. In this embodiment, a 5 x 4 pixel chunk like the one 
shown in Rgure 14 is stored as a single unit (See Rgure 
1 6). In this way the volume of data is compressed to 
1/20 of its former size. The subsequent processing to 
extract the mark (or rather, to extract the spatial relation- 
ship of the marks) can easily be executed using a small 
volume of data. The size of the chunks into which the 



data should be compressed should be determined so 
that a mark can fit into a single chunk. Alternatively if a 
mark spans several chunks, one of the chunks (for 
example, the one containing the largest portion of the 
5 mark) will receive a "1 " to indicate that a mark has been 
found. 

The internal configuration of mark location detec- 
tion unit 15, the device which executes the processing 
discussed above, is shown in Figure 17. Unit 15 con- 

10 sists of window unit 1 5a, which measures 5x4 pixels, 
and decoder circuit 15b. which receives the pixel data 
(1/0) detected by unit 15a and determines whether they 
are in a given arrangement. 

More specifically, as is shown in Figure 18. each Qy 

75 in the window consists of a flip-flop. The window has 
four rows in each of which five flip-flops are connected 
in series. A row of flip-flops corresponds to the pixel 
data (1/0) stored on a single line in the line buffer which 
serves as storage device 14. The data are input into the 

20, flip-flops, starting with the head flip-flop, in order from 
the first pixel. Each flip-flop simultaneously receives a 
clock (CLK) signal. Once synchronized, it transmits the 
data to the next flip-flop. 

When clock signals are input one by one, the input 

25 of data to the flip-flops assumes the same form as scan- 
ning, pixel by pixel in the scanning direction. When the 
final pixel data for a given line (1/0) have been input, we 
begin to input data to the head flip-flop of the.next line 
down. This is identical to moving down one pixel in the 

30 feed direction. Consequently, the output of the flip-flops 
when five pixels* worth of data have been input will be 
as shown in the drawing and will be equivalent to the 
window pictured in Figure 15. The output MWQy (1 = 0 
through 3. j = 0 through 4) of each flip-flop is sent to 

35 decoder circuit 15b. 

Decoder circuit 15b. which is pictured in Figure 19, 
consists of a 20-input AND element. When a mark is 
input, the Input terminals constituting white pixels have 
their input reversed. Thus when black pixels "1" are 

40 lined up in a given shape, all the inputs of the AND ele- 
ment which constitutes the decoder circuit go to "1 ". and 
the HIT output of the decoder circuit goes to "1". If the 
value of a single pixel is different from the that of the oth- 
ers, one of the inputs to the AND circuit will be "0"^ and 

45 the HIT output of the decoder circuit will go to "0". Based 
on the output of decoder circuit 15b. data representing 
the locations of all the valid marks 2 are stored in stor- 
age device 16. If we are to map out the internal data 
stored in device 16 when the pattern has been detected. 

50 they would be arranged as shown in Figure 20. Each 
square in the drawing corresponds to a 5 x 4 pixel chunk 
of the image data captured when the document is 
imaged. 

As stated above, mark location detection unit 15 
55 outputs a compressed binary image in which portions 
where marks 2 are found go to "1" and pixels of other 
colors are expressed as "0". This image is stored in 
storage device 16. which in this embodiment also con- 
sists of a given number of line buffers. Since the speci- 
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fied pattern is detected based on this compressed 
binary Image, there is no need to store all the image 
data representing the document- Only a few dozen lines 
of buffer space are needed to store the data needed for 
subsequent processing. 

Pattern location matching unit 1 7, the device which 
detects pattern 1, is configured as shown in Figure 21. 
It has a window 1 7 of a given size, which it uses to scan 
the mark data (1/0) stored in device 1 7. 

In this example, pattern 1 consists of a number of 
marks 2 placed along the outlines of two squares, as 
shown in Figures 1 and 2. For this reason it Is desirable 
to extract simultaneously all the "rs stored in device 16, 
which can be seen in Figure 20. Leaving a margin one 
chunk wide around the outer square on which the marks 
are placed, we use a window of 10 x 10 chunks, as 
shown in Figure 22. The hatched portions of the draw- 
ing (inner square and outer square W2) are the 
regions where data will actually be collected. 

The actual configuration of window unit 17a is 
shown In Figure 23. Ten rows of flip-flops with ten serial 
inputs and parallel outputs are provided. Data in a given 
line buffer of device 16 are input into the corresponding 
row flip-f bps in order starting with the head flip-flop. 
The; operation Is identical to that of the aforementioned 
5 X 4 joixel window unit 15a in mark location detection 
unit 15 Data synchronized by a clock signal CLK are 
transmitted In order. The output of the flip-flops at any. 
given mpment, as shown in Figure 24' functions as a 10 
X 10_ijhit window. 

Tfie units in the output which correspond to the 
hatched portions of Figure 22 are AWQ33 through 36. 
; 43. 46^53, 56 and 63 through 66 (inner square W^) and 
' AWQUI through 18. 21. 28, 31, 38. 41. 48, 51, 58, 61. 
68. 71. ' 78 and 81 through 88 (outer square Wg). These 
data'cah be used to detect pattern 1 . 
/ i The data stored in the units of outer square W2 (the 
^ou4>Lrts of the specified flip-flops stated above) are sent 
to latching circuit 17b; those stored in the units of inner 
square (the outputs of the specified flip-flops stated 
above) are sent to latching circuit I7c. These circuits 
latch to conform to the working of window unit 1 7a. 

LatQhIng circuit 17b consists of 28 D latches (flip- 
flop groups wItK parallel input and parallel output), as 
showp in Figure 25. These latches maintain the Input 
data ;as they are when the latch signal is received. In 
this eiample. AWQ15 is the head of the unit outputs 
constituting the window in Figure 24. The units which 
correspond to the, location of outer square W2 are 
arranged in order in a clockwise circuit. Accordingly, as 
Is^showvn In Figure 25. AWQ15 Is input to the head D 
latch and AWQ14 is input to the final D latch, 
r^^^* ^Similarly, latching circuit 17c consists of 12 D 
, Jatches as shown in Figure 26. AWQ35 is the head of 
I the unit outputs constituting the window In Figure 24. 
YThe units which con-espond to the location of Inner 
square Wi are arranged in order in a clockwise circuit. 
AWQ34 is input to the final D latch. 
/a As Is shown in Figure 21 . tfie output of latching clr- 
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cuit 17c is sent to address decoder circuit 17d. The 12- 
brt data latched In circuit 17c are used as address data 
to access the con-esponding portion of dictionary stor- 
age unit 17e. Dictionary data for 0 deg., 90 deg.. 180 
deg. or 270 deg. on the outer square are output to com- 
parison circuit 17f. The 28-bit data latched in latching 
circuit 1 7b are sent to comparison circuit 1 7f. where the 
data are compared bit by bit The result, match or no 
match, is transmitted to counter circuit 17g. 

Generally when a document is copied, it must be 
placed parallel to the sides of the glass. Although the 
document may be parallel, it sometimes happens that it 
has been placed on the glass with its top and bottom or 
left and right sides reversed. Thus a specified pattern on 
a document may be oriented in any one of four direc- 
tions, as shown In Figures 27 (A) through (D). When the 
marks 2 constituting pattern 1 are stored In a specified 
area of storage device 16. the data latched in latching 
circuits 17b and 17c by way of latching unit 17a are 
an-anged in four ways, as shown in Figure 28. 

The data arrangement is stipulated in this way. and 
the relationship of the data latched by a mark on inner 
square at a given angle of rotation to tiie data 
latched by a mark on outer square Wg is one to one. If 
the marks have been arranged appropriately, the data 
will be different at the four angles of rotation shown in 
Figures 27 and 28. If we find a match while trying to 
determine which data arrangement a certain region (in 
this example, inner square , which has fewer data) 
corresponds to, there is a high probability that we have; 
found the specified pattern. Since we now know its 
angle of rotation, we can use the data latched from 
outer square W2 as reference values (I.e.. data which, 
are valid if this is the specified pattern). We compare,^ 
these reference values with the actual data obtained; 
from outer square W2. A high coincidence indicates that 
the pattern has been recognized. Arrangement match- 
ing unit 1 7 uses this same principle. 

In this embodiment, efficient use is made of the 
data from inner square W^ and outer square Wg. As disr. 
cussed atxjve. the less voluminous data from inner, 
square Wi are used as addresses. The data strings Jnl 
the units in Figure 28 are stored at specified addresses -i 
In dictionary storage unit 1 7e. 

The data latched by the circuit for the inner square. 
1 7c, are used as addresses. If there Is a match with one 
of the four addresses shown in Figure 28. the ,data 
stored in dictionary storage unit (ROM) 1 7e which corre- 
spond to this address are read out and compared by , 
comparison circuit 17f. By checking whether there is an 
address which corresponds to the mark data on Inner 
square W^, we determine whether the marks we have 
found are arranged in pattern 1. Comparing the mark 
data on outer square W2 with reference values which 
have been read out also allows us to determine whether 
the marks we have found are ananged In pattern 1 . 

Comparison ciraiit 17f is composed of D latch unit 
Di, which stores the data collected from the outer 
square; D latch unit D2, which stores the dictionary data 



f%NSDOCID: <EP_:0751475A2 t > 



10 



19 



EP 0 751 475 A2 



20 



(i.e.. the reference data) it receives from dictionary stor- 
age unit 17e for the outer square at specified angles of 
rotation; a dual-input coincidence circuit CC, which 
compares the data stored in latches and D2; and par- 
allel input-series output shift register SR, which receives 5 
each output of coincidence circuit CC and outputs it in 
the same order. Units Di. D2. CC and SR each have 28 
stages to correspond to the number of data (28) com- 
prising outer square W2. 

When this configuration is used, the data transmit- 
ted by latching circuit 1 7b and those received from dic- 
tionary storage unit 17e (the reference data) are stored 
in their respective latching units, Di and D2. They are 
input into coincidence circuit CC in response to a spec- 
ified timing signal. If the reference data and the data 
indicating the actual locations where the marks are 
extracted are identical, the output of coincidence circuit 
CC will go to "1 If they are not, it will go to "0". The 
result of this determination is stored in the correspond- 
ing shift register SR. A shift clock is applied to shift reg- 
ister SR, and the data stored in it are input to counter 
circuit 1 7g in order, bit by bit. 

Counter circuit 17g counts the number of data 
which are "1 If all the data acquired from outer square 

coincide with the reference data stored in the dic- 
tionary, the 28 bits output by the shift register will all be 
T. If none of them coincide, the 28 bits will all be "0". 
The higher the count value, then, the higher the good- 
ness of fit of the data to the specified pattern. In this 
example, the count value is the goodness of fit of the 
detected pattern with respect to the specified pattern. 
This count value is output by way of goodness output 
unit 18 to the copy machine. 

Based on the goodness of fit which it obtains, the 
copy machine decides if the document being processed 
is non-reproducible. If it determines that the document 
Is non-reproducible, it executes the specified process- 
ing to prevent copying. Alternatively, the image process- 
ing device decide if the document is non-reproducible. 
When the goodness of tit exceeded a threshold value, it 
would judge the document to be non-reproducible and 
output a signal indicating detection. 

Figures 30 and 31 show another image processing 
device which is an ideal embodiment of this invention. 
This device is based on the ideal embodiment we have 
been discussing, but its pattern recognition is more reli- 
able. Our second ideal embodiment, image processing 
device 10', has. in addition to the devices seen in the 
first embodiment which calculate the goodness of fit of 
the shape, the capability of calculating the goodness of 
fit of the color of the marks. Based on both the shape 
and color data, a cumulative judgment can be made as 
to whether the specified pattern has been found. The 
pattern 1 and marks 2 detected by this device are the 
same as those used in the first embodiment. Marks 2 
are printed In yellow ink In the dot mark shown in Figure 
14. Pattern 1 consists of the arrangement of marks 2 
shown in Figure 1 . 

In our discussion of the actual configuration of this 



device, components which have the same function that 
they had in the first eml^odiment (including those with 
more output ports) are labeled with the same numbers: 
we will not discuss them further. We begin our discus- 
sion of points which differ from the first embodiment with, 
input unit 12, whose output is transmitted to both binary 
unit 13 and third memory unit 19. Unit 19 stores the 
untreated RGB data as multiple values. It outputs to 
extraction unit 20 the data required to extract the areas 
where the marks are found. 

Extraction unit 20 receives the output of mark loca- 
tion detection unit 15. needed to calculate the goodness 
of fit of the shape. When the marks 2 which constitute 
pattern 1 have been detected (i.e., when the HIT output 
of detection unit 15 is "1"). the density data (RGB) for 
the pixels in a 5 x 4-pixel region around the location of a 
mark (i.e.. the region where a mark 2 is found) are read 
out of third memory unit 19 and transmitted to unit 21, 
which calculates their average density. The 5 x 4-plxel 
region which is extracted includes the pixels which sur- 
round the mark. It would, of course, also be possible to 
transmit to calculation unit 21 only the density data for 
pixels which actually constitute the mark in the 5x4 
pixel region (the black pixels in Figure 14). 

Calculation unit 21 obtains the average values of 
the R, G and B densities it receives from extraction unit 

20 and outputs them to density matching unit 22. 

Matching unit 22 matches the averages of the R. G 
and B densities which it receives from calculation unit 

21 against previously recorded color data to obtain the 
goodness of fit of the color. An example of an algorithm 
which could be used to calculate goodness of fit would 
be to record the color data in the form of the member- 
ship functions pictured in Figure 31 . Fuzzy matching 
could then be executed for each component to obtain its 
goodness of fit, and the three values could be averaged 
to obtain the goodness of fit of the color. In the example 
in Figure 31 . the goodness of fit of the R density is 0.8; 
that of the G density is 1 .0; and that of the B density is 
0.8. The overall goodness of fit. which we obtain by 
averaging these values, is 0.87. 

Since the pattern described above is composed of 
a number of marks, the overall goodness of fit of the 
color can be determined by calculating the goodness of 
fit of the color of each mark as described above and 
making appropriate use of the data for all the marks 
obtained in this way. For example, once the goodness of 
fit had been found for the color of each mark, we could 
execute any of a variety of calculations and statistical 
operations, such as obtaining the average value of the 
goodnesses of fit or their distribution. Or instead of 
obtaining the goodness of fit of the color after that of the 
marks, we could obtain the goodness of fit of each of the 
R. G and B components of all the marks (as an average, 
say). By finally averaging these three values, we could 
obtain the overall goodness of fit of the color 

Based on the goodnesses of fit of the shape and 
color which It receives, the copy machine makes an 
overall judgment as to whether the document is non- 
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reproducible. It would, of course, also be possible for the 
image processing device to handle this overall judg- 
ment. With this configuration, even if the goodness of fit 
of the color or the shape of the pattern is reduced by 
tampering, the recognition rate will not be affected. 5 

Rgure 32 shows an image processing device which 
is our third ideal embodiment of this invention. This 
embodiment differs from the two described above in that 
the judgment processing is based on a single color sig- 
nal. Image processing device 10" lacks the mark color 10 
extraction unit 13b and the AND element 13c which are 
components of binary processing unit 13 in image 
processing device 10. the first embodiment; its unit 13 
consists only of mark shape extraction unit 13a. Since 
the rest of the configuratipn of device 10" is basically 75 
identical to that of the previous embodiments (except for 
the fact that its image input unit 12 processes only one 
signal), the components have been given the sane num- 
bers and are not discussed here. 

In this embodiment, if yellow is chosen for the 20 
marks, the B signal will be input into image input unit 12 
if the format is RGB; the Y signal will be input if the for- 
mat is YMC; and the b signal will be input if the format is 
Lab 

Image input unit 1 2 transmits the single color signal 25 
it receives to binary processing unit 13*. Regardless of 
whether the signal is B. Y or b. its numerical value will 
be low. If the reference values in mark shape extraction 
unit 13a are made equally small, processing unit 13b 
will generate a binary image in which all pixels of the tar- 30 
get color are "1". This image is transmitted to storage 
device 14. This device processes the binary image data 
stored in device 14 just as the first embodiment did and 
outputs the goodness of fit. 

With this configuration, the same image processing 35 
device 10" can be used in a variety of copy machines 
which use different formats for the image data (RGB. 
YMC or Lab), and rt will extract the pattern accurately. 

Since only one color signal is used to make a judg- 
ment, the configuration of the circuit is simplified. In 4o 
copy machines which scan the image multiple times or 
those in which the data are received in the same order 
that they are acquired, the color signal used for the judg- 
ment can be acquired first and sent to processing 
device 10". This will allow the subsequent judgment of 45 
whether the data contain the specified pattern to be 
made relatively quicWy (before the image is completed). 
The copy processing can be halted before the image is 
connpleted, eliminating the possibility that a completed 
image will be erroneously output. so 

Figures 33 and 34 show an example of a device as 
described above which has been installed in a copy 
machine. As shown in the drawing, the light emitted by 
lamp 32 is reflected by document 31 . which is loaded on 
glass 30. The reflected light passes through optical sys- 55 
tem 33 and enters CCD 34, which is an image sensor. 
Here the image on the document is read. Lamp 32, as 
well as the flat mirror and other conponents which com- 
prise optical system 33. moves at a specified speed as 



it scans the document. Line by line. CCD 34 reads the 
specified portion of document 31 . and it transmits the 
image data (R/G/B) to signal processing unit 35. 

Signal processing unit 35. which is pictured in Fig- 
ure 34. has an ordinary color processing circuit 36 and 
device 37, which is the image processing device dis- 
cussed above as a feature of this invention. The afore- 
said image data are transmitted in parallel to color 
processing circuit 36 arxd to image processing device 
37. If device 37 is realized as the third ideal embodi- 
ment, either one of the RGB signals or a specified out- 
put signal from color processing circuit 36 (for example, 
the Y signal) can be input. 

Color processing circuit 36 analyzes the colors into 
their various components, magenta (M), cyan (C), yel- 
low (Y) and black (Bk), and outputs these data to print- 
ing device 38. The image is actually scanned four times. 
One of the aforesaid four components (M, C, Y. or Bk) 
resulting from the first scan is output to laser driver 39, 
which is located on the input side of printing device 38. 
This causes a laser beam to be projected onto a speci- 
fied location on light-sensitive drum 40. When the fourth 
scan has been completed, the copying operation is exe- 
cuted on the copy paper, and the finished copy 41 is out- 
put. Since the actual mechanism which executes the 
copying operation is identical to that of existing copiers, 
we shall omit an explanation of its action. 

While the aforesaid image processing circuit 36 is 
processing the signals and in parallel with this process- 
ing, image processing device 37 processes the RGB 
signals which it receives (or one of them) to obtain the 
goodness of fit of the shape (the first and third embodi- 
ments) or of both the shape and the color (second 
embodiment). The goodness of fit which is obtained is 
transmitted to determination unit 42 in the PRC. 

Based on the goodness of fit which it receives, 
determination unit 42 determines whether the specified 
pattern has been found on the document being proc- 
essed. If the goodness of fit is high, unit 42 concludes 
that this is a non-reproducible document on which the 
specified pattern is printed. It then generates a control 
signal to halt the output of the aforesaid laser driver 39 
or transmits a control signal to color processing circuit 
36 to cause it to execute its various routines for situa- 
tions when copying is prohibited, such as making the 
entire screen go black. 

If image processing device 37 did not output good- 
ness of fit, but instead makes the determination itself, 
the aforesaid determination unit 42 would be unneces- 
sary. The output of image processing device 37 (a sig- 
nal indicating that the specified pattern has been 
detected) could be transmitted to laser driver 39 or 
image processing circuit 36. 

In the embodiments discussed above, the device is 
installed in a copy machine; however, this invention is 
not limited to this application, but could be applied as 
well in a color scanner, a color printer, a fax machine, a 
device to transmit communications, or a variety of other 
devices. 
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To give an example, a scanner employing this 
invention might have the configuration pictured in Figure 
35. The scanner comprises three main components: 
input unit 50, control unit 51 and output unit 52. Input 
unit 50 scans document using light from a light source. 5 
The light reflected (or transmitted) by the document is 
detected by a photoelectric converter element such as a 
CCD, a photomultiplier, or a photodiode. This light is 
converted to an electrical signal which is transmitted to 
control unit 51. Control unit 51 amplifies the electrical 
signal from the input unit and executes various kinds of 
image processing, such as the correction of specified 
tones or contour enhancement, and transmits the cor- 
rected signal to output unit 52. 

Output unit 52 transforms the data as needed, 
based on the signal it has received from control unit 51 , 
and outputs them to the specified output device. When 
the scanner and printer are separate devices and the 
scanner does no more than read the document, the 
data will need to be sent from scanner to printer. It will 
thus be necessary that the image data which the scan- 
ner has read be stored temporarily in a memory device 
(an output device). Whatever processing is required to 
write the data will be executed. 

If the output device is a printer or the like (which is 
installed in the same device), the signal must be con- 
verted from an electrical to an optical signal, and the 
processing must be executed to convert the signal in 
order to write the data on a paper medium (a photosen- 
sitive material). Since commonly available devices may 
be used as the actual components, we shall omit a 
detailed discussion of them here. 

With this invention, an image processing device 53 
is provided. The signals representing the image data 
which are received from the said input unit 50 are input 
into image processing device 53 as well as into control 
unit 51 . Image processing device 53 can employ any of 
the processors in the embodiments discussed above 
which obtain the goodness of fit with respect to the 
specified pattern. 

Image processing device 53 executes the specified 
processing on the image data which it receives. It 
obtains the goodness of fit. which is then used to detect 
the specified pattern. Device 53 sends the goodness of 
fit (or a "prohibit output" signal) to control unit 51 . Based 
on this goodness of fit, control unit 51 makes the final 
judgment as to whether the specified pattern has been 
found. If it concludes that the pattern has been found, it 
halts the signal output to output unit 52. A "prohibit out- 
put" signal could also be sent to input unit 50 or output 
unit 52. 

If in this embodiment image processing device 53 
makes the final judgment as to whether the specified 
pattern has been found and concludes that the pattern 
has been detected, it can output a "prohibit output" sig- 
nal itself. In this case, control unit 51 will respond to this 
signal by executing the aforesaid specified processing 
to interrupt the copying or printing process. 

Figure 36 shows how this signal processing device 
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could be used in a printer. The image data (in the form 
of electrical signals) are sent to input unit 54 either 
directly from the scanner or through a medium such as 
a storage device. Control unit 55 executes the process- 
ing necessary to convert the image (i.e.. it converts the 
data to the form required by the output mechanism) . 
Output unit 56 executes the processing required to con- 
vert the signal from electrical to optical, and the image 
data are reproduced on a photosensitive material. 

If the scanner described above does not have an 
image processing device 53 with a means to halt its 
operation when a specified pattern has been detected, 
the image data will be read even if the document is one 
which should not be read. 

The output signal from control unit 55 is sent to 
image processing device 57 (identical to the aforesaid 
image processing device 53), which executes the spec- 
ified image processing, obtains the goodness of fit indi- 
cating the degree of resemblance to the specified 
pattern, and executes processing to detect the pattern. 
If the method entails using the goodness of fit, this value 
is transmitted to control unit 55. Unit 55 uses the good- 
ness of fit to make the final determination as to whether 
the specified pattern has been found. If it determines 
that it has, it halts data output to output unit 56. An 
"operation prohibited" signal could also be sent directly 
to output unit 56 to cause that unit to halt its operation. 

If in this embodiment image processing device 57 
makes the final judgment as to whether the specified 
pattern has been found and the pattern has been 
detected, it can output an "operation prohibited" signal 
to control unit 55. This will halt data output from control 
unit 55 to output unit 56. Alternatively, it can send an 
"operation prohibited" signal directly to output unit 56 to 
halt its operation. 

This invention has the following effects. As stated 
above, the image processing device and method of this 
invention generate a binary image in which the marks 
on a document are made to stand out from their back- 
ground. It generates this image by executing threshold 
processing on a single color signal. Once the pattern 
has been generated, the pattern can be extracted. The 
processing is executed by a comparator consisting of a 
specified number of bits. This allows the binary circuit to 
be simplified and reduces the size arxi cost of the sys- 
tem. 

The marks are detected by extracting only a speci- 
fied color. If the marks of the pattern are printed along 
with other marks with a low density of the chosen color, 
executing threshold processing on a single color signal 
will point up only the marks in the pattern so that they 
can be detected. The recognition rate will not be 
affected by the presence of extraneous marks of 
another color. An uninformed person will not know 
which of the different colored marks form the pattern, 
and it will be more difficult to tamper with the document. 

If the color signals are transmitted in the order of 
their surfaces, the signal which is transmitted first can 
be selected to be used for pattern recognition. This will 



EP 0 751 475 A2 



15 



20 



25 



30 



35 



40 



45 



SO 



55 



13 



BNSDOCID: <EP 0751475A2J_> 



25 



EP 0 751 475 A2 



26 



allow pattern recognition to be completed before the 
entire image has been transmitted. The generation of 
the image can be interrupted at that point, and en-one- 
ous operation can be minimized. 

With one configuration described, the goodness of s 
fit of the shape and that of the color are obtained sepa- 
rately, so the recognition rate will remain high even 
when the shape of the marks or their color has been 
altered. In actual situations, it frequently happens that 
the color of some of the marks varies as a result of a io 
printing error or dirt on the document As can be seen in 
the embodiments, density data can also be used when 
the image is binarized to make the marks discernable. 
To allow for some color variation of the type mentioned, 
the threshold value must be set low to allow a range of 75 
colors to pass. The threshold value used to extract a 
single color signal must also be set low. In this case it is 
possible that false positives will be identified when the 
marks are all of different colors. However, if we make 
the threshold for the goodness of fit of the color very 20 
strict but also make a cumulative judgment based on the 
colors of all the marks, we will be able to distinguish 
accurately between printing errors and marks which are 
actually of different colors. This will enhance the recog- 
nition rate. 25 

Using yellow as the color for the marks allows pat- 
tern recognition to be executed in any image format, 
whether RGB. Lab or YMC. This allows the device to be 
used in any type of copy machine. It also makes it more 
difficult to tamper with the pattern, as yellow is difficult 30 
for the human eye to discrimirate. 

With the other configurations described, the speci- 
fied pattern consists of a number of marks placed along 
the outlines of several shapes. Thus there is no need to 
use a particular shape for the area to be extracted, and 35 
it will be more difficult for someone to discern that there 
is a pattern. The pattern will not need to be scanned 
separately to extract the marks, and both the marks and 
the pattern they form can be extracted on a single scan. 

An uninformed person will not know which portion 40 
of the marks constitutes the pattern, and so will find it 
more difficult to alter or deface it. Even if the document 
is defaced, it is unlikely that the portion defaced will be 
the specified pattern, so the recognition rale will not be 
affected. The marks are expressed only as their coordi- 45 
nates, so no feature analysis is required. Very small 
marks can be used, which will make it difficult for a per- 
son even to notice that the marks are there. This will 
also make it more difficult to tamper with the pattern. 
Making the marks smaller allows them to be found so 
faster. It also allows us to use a smaller and less costly 
detection circuit. 

The marks are placed along the outline of a shape 
or shapes. Identical marks in other locations will not be 
used to detect the pattern. This allows us to use marks ss 
which are not on the outline of the shape as a dummy 
pattern to further reduce the likelihood of tampering. 

When such an image processing device is installed 
in a copy machine, scanner or printer, it will reliably pre- 



vent the output of copies of documents which may not 
legally be copied, such as k>ank notes or negotiable 
securities. (The copying process itself will not be exe- 
cuted, or an image which is different from the original 
non-reproducible document will be copied and output.) 
It will also halt the reading or printing of the document. 

Claims 

1 . An image processing method to detect a particular 
mark of a givai shape and color in a received 
image data of a color image, comprising a step of 
detecting said particular mark shape using only a 
single color signal out of a plurality of color signals 
conprising said received image data of said color 
image. 

2. An image processing method according to claim 1 , 
wherein said single color signal is received first 
among said plurality of color signals which are 
received successively 

3. An image processing method according to claim 1 . 
further comprising a step of extracting a mark color 
by performing threshold processing on said single 
color signal and other color signals data whose 
color density is within a given range. 

4. An image processing method according to claim 3. 
further comprising the steps of; 

calculating an average color density of said 
particular mark; 

calculating a goodness of fit between said aver- 
age color density of said particular mark with a 
given reference color value; and 
outputing a total goodness of fit for said partic- 
ular mark shape and said average color den- 
sity. 

5. An image processing method according to claim 1 , 
wherein said single color signal is yellow; 

6. An image processing method to detect a particular 
pattern constituted by a plurality of particular marks 
of a given shape and color in a received image data 
of a color image, comprising a step of: 

detecting the shape of said particular marks 
using only a single color signal out of a plurality 
of color signals comprising said received image 
data of said color image. 

7. An image processing method to detect particular 
pattern according to claim 6 wherein said single 
color signal is received first among said plurality of 
color signals which are received successively, 

8. An image processing method to detect particular 
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pattern according to claim 6. further comprising a 
step of extracting a mark color by performing 
threshold processing on said single color signal and 
other color signals whose color density is within a 
given range. 

9. An image processing method to detect particular 
pattern according to claim 8. further comprising the 
steps of: 

calculating an average color density of said 
particular marks; 

calculating a goodness of fit between said aver- 
age color density of said particular marks with 
a given reference color value; and 
outputing a total goodness of fit for said partic- 
ular marks shape and said average color den- 
sity. 

10. An image processing method to detect particular 
pattern according to claim 6, wherein said single 
color is yellow. 

11. An image processing method to detect particular 
pattern according to claim 6. said plurality of partic- 
ular marks being located on peripheries or a 
number of given patterns, comprising a step of: 

obtaining an angle of orientation of said partic- 
ular pattern based on data representing said 
particular marks on a periphery of one of said 
pattern; 

extracting a reference pattern based on said 
angle of orientation out of said number of given 
patterns; and 

comparing said particular pattern with said 
extracted reference pattern to calculate a 
goodness of fit. 

12. An image processing device to detect a particular 
mark of a given shape and color in a received 
image data of a color image, comprising: 



rality of color signals other than said one 
extracted by said mark shape extracting unit, 
and detect if a color density of said color sig- 
nals are within a given range; and 
5 an AND element receiving outputs of said mark 

shape extraction unit and said mark color 
extraction unit. 

14. An image processing device according to claim 13, 
10 further comprising; 

a third memory unit parallel to said binary 
processing unit to store said received image 
data; 

75 a mark region extraction unit to extract said 

particular mark identified by said mark location 
detecting unit; and 

an average density calculation unit to calculate 
an average color density of said particular 
20 mark; and 

a density matching unit to detect a goodness of 
fit of said particular mark against a given color 
density. 

25 15. An image processing device according to clam 12. 
further comprising: 

a second memory unit to store a mark location 
data output from said mark location detecting 
30 unit; and 

a pattern matching unit to match a pattern con- 
sisting of said particular marks against a given 
reference pattern consisting of a number of 
marks based on said mark location data out- 

35 put. 

16. A copy machine having a reading unit to read an 
original document to generate an image data, a 
color processing unit to convert said image data of 
40 said reading unit, and a printing unit to print an 
image output from said color processing unit, com- 
prising: 



a binary processing unit having a mark shape 

extracting unit to execute threshold processing 45 

on one of a plurality of color signals for a color 

image to generate a binary image output; 

a first memory unit to store said binary image 

output by said binary processing unit; and 

a mark location detecting unit to read out said so 

binary image output from said first memory 

unit, and match a location of said particular 

mark. 

1 3. An image processing device according to claim 1 2, 55 
wherein said binary processing unit further com- 
prises: 

a mark color extraction unit to extract said plu- 



an image processing device to detect a partic- 
ular mark of a given shape and color in a 
received image data of a color image; 
a binary processing unit having a mark shape 
extracting unit to execute threshold processing 
on one of a plurality of color signals for said 
color image to generate a binary image output; 
a first memory unit to store said binary image 
output by said binary processing unit; and 
a mark location detecting unit to read out said 
binary image output from said first memory 
unit, and match a location of said particular 
mark. 

wherein said image data is input parallel into 
both said image processing device and said color 
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processing unit, said image processing device 
detects said particular mark during a given scan 
and decides if said document being processed is 
one which may not legally be reproduced, and 
transmits a control signal to said printing unit to 5 
control a copying process. 



decides if said image data being processed is one 
which may not legally be reproduced, and transmits 
a control signal to said printing unit to control a 
printing process. 



17. A scanner device having a reading unit to read an 
original document to generate an image data, a 
control unit to process said image data connected 10 
to said reading unit, and an output unit to output 
said processed image data to a output device con- 
nected to said scanner, comprising: 



an image processing device to detect a partic- is 
ular mark of a given shape and color in a 
received image data of a color image; 
a binary processing unit having a mark shape 
extracting unit to execute threshold processing 
on one of a plurality of color signals for said 20 
color image to generate a binary image output; 
a first memory unit to store said binary image 
output by said binary processing unit; and 
a mark location detecting unit to read out said 
binary image output from said first memory 25 
unit, and match a location of said particular 
mark. 



wherein said image data is input parallel into 
both said image processing device and said control 30 
unit, said image processing device detects said 
particular mark during a given scan and decides if 
said document being processed is one which may 
not legally be reproduced, and transmits a control 
signal to said control unit to control a scanning 35 
process. 



18. A printer having a control unit to process an input 
image data, and printing unit to print said proc- 
essed image data, comprising: 40 



an image processing device to detect a partic- 
ular mark of a given shape and color in a 
received image data of a color image; 
a binary processing unit having a mark shape 45 
extracting unit to execute threshold processing 
on one of a plurality of color signals for said 
color image to generate a binary image output; 
a first memory unit to store said binary image 
output by said binary processing unit; and so 
a mark location detecting unit to read out said 
binary image output from said lirst memory 
unit, and match a location of said particular 
mark. 

55 

wherein said input image data is input paral- 
lel into both said image processing device and said 
control unit, said image processing device detects 
said particular mark in said input image data and 
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FIGURE 11(A) 
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FIGURE 12(A) 
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FIGURE 13(A) 
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FIGURE 13(B) 
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